Education
"I was thinking about the future and I thought, 'Am I willing to risk losing data?'"
12 Fargate spot instances with 4 CPUS and 24GB RAM each
~100 tables stored in CockroachDB
Scaled through COVID-19 demand spikes
Founded in 2018, MyMahi is a fast growing digital education platform based in New Zealand. The MyMahi platform helps students set goals and commit to activities that will contribute to their wellbeing outside the classroom and prepare them to enter the workforce. 40% of the secondary schools in New Zealand have already adopted MyMahi and the company is preparing to expand their business into Australia.
Building an application focused on the rhythms of student life requires the flexibility to easily scale up and down to match periods of high and low activity. MyMahi also needs an ACID compliant database that can easily be maintained by a small development team. For these reasons and more MyMahi elected to use CockroachDB instead of more familiar databases like Postgres or MySQL.
This post will explore how CockroachDB benefits MyMahi and how MyMahi built a scalable serverless GraphQL backend with CockroachDB and spot instances.
When lead engineer, Stefan Charsley, joined MyMahi there was an application already in place. It was built by a contractor on top of PHP and, at first glance, Stefan knew that it was not worth keeping. The application was not architected for scale. And it would be difficult to maintain a system built by others. Both of which are problems for a small engineering team that needs to focus on feature development, not maintaining infrastructure.
What Stefan and his team of developers wanted to build was an application that would be easy for end-users to adopt, and easy for a small team of developers to maintain. In order to accomplish this he knew he needed to build the application architecture from scratch.
“I was thinking about our future and I thought, ‘how much harder will it be to scale Postgres compared to CockroachDB.’ And I thought, ‘Am I willing to risk losing data?’” -Stefan Charsley, Senior Software Engineer
Bootstrapping a startup means that cost has an influence on technical decision making. But cost does not need to stand in the way of innovation.
The engineering team from MyMahi has years of experience trying to scale and maintain Postgres. They knew they wanted to build MyMahi’s application architecture with cutting edge tools that could take better advantage of the cloud and set MyMahi up to grow and scale with less labor.
In their search for the right database they established the following requirements:
CockroachDB serves as the general-purpose database for MyMahi’s digital education and personal growth platform that has already been adopted by 40% of the secondary schools in New Zealand. Ultimately, MyMahi chose CockroachDB over familiar solutions like Postgres or MySQL for practical and philosophical reasons:
They needed to be able to easily scale up and down to match the bursty pattern of engagement with student schedules.
They see opportunities to scale the business to different countries and the simplicity of scale in CockroachDB is a huge improvement over the manual scale operations in Postgres.
They want to maintain uptime and be able to upgrade the server without having to bring down the database. Because of CockroachDB’s multi-active availability design you can do a rolling upgrade in which one node is upgraded at a time without interrupting operations.
They wanted Postgres compatibility for comfort and productivity. CockroachDB speaks the Postgres dialect. Stefan says, “It’s basically like working with a cloud-native Postgres.”
They don’t like the ‘leader, follower’ architecture of Postgres and MySQL. They liked the fact that CockroachDB is multi-active and all the nodes are doing work. The multi-active model ensures that clusters are always totally consistent and tolerant to failures.
Philosophically, MyMahi is intentionally creating a culture of innovation. The product itself is attempting to innovate on the traditional structure for learning and personal growth. The application architecture that supports the product is embracing the freedom to innovate on the traditional architecture that the engineers have seen at prior companies.\
MyMahi is using the core version of CockroachDB and has it deployed in AWS with 12 Fargate spot instances with 4 CPUS and 24GB RAM each.
They currently have around 100 tables stored in CockroachDB. Most of the data being stored by CockroachDB is student data that must be consistent. This student data includes goals set by students, extracurricular activities, written reflections, a resume builder tool, and some analytical data like page views and user actions.
Here is a look at the primary components of the architecture:
For an extremely detailed description of the application architecture please jump to this technical blog post.
Here is how the application architecture functions: The platform is built in the AWS cloud. User requests are handled by Route53 DNS resolution. Requests are sent to Cloudfront (the AWS content delivery network (CDN)). Static content like image assets are delivered from an S3 bucket. Other requests are handled by API Gateway. Each API Gateway request is handled by Lambda, which is the serverless platform in AWS that can run stateless application code so nobody has to think about where the code is running. The Lambdas have a 30-second timeout, so longer requests are sent to an SQS queue for processing by async Lambdas, which have a 15-minute timeout. Shorter requests are handled synchronously by fetching data and performing business logic. Some user content is in another S3 bucket, while other data is stored in a CockroachDB cluster.
MyMahi deploys CockroachDB on AWS Fargate spot instances, which offer AWS spare capacity at a low cost. Fargate is another serverless platform, except instead of just running small functions, it runs long-lived containers. It’s important to note that MyMahi can use spot instances only because of CockroachDB’s resiliency to node failure. They would not choose to use spot instances with a less resilient database. This is because AWS can reclaim (i.e destroy) instances with an alarmingly short 2 minute notification.
Each component of this architecture can respond to increases in traffic, and because Fargate uses usage-based pricing, costs can remain low.
CockroachDB unlocked some interesting technical decisions in the application architecture for MyMahi. One being the ability to pair the database with AWS Lambda (the other being the use of GraphQL). AWS Lambda allows for seamless scaling at the application layer which pairs well with the seamless scale of CockroachDB.
MyMahi uses Lambda with Node.js to handle bursts of traffic. The Lambda functions have all the code for connecting to and querying CockroachDB. Node.js is used to implement the Lambda functions. For code snippets and extensive detail on MyMahi’s use of Lambda check out this blog. What’s important is that the use of Lambda allows MyMahi to keep costs down while student engagement with the platform is low and then it can quickly scale up when thousands of students jump on the platform at the same time.
Initially, the use of GraphQL with CockroachDB was an experiment. But it worked out well and has become a staple part of their architecture. GraphQL allows MyMahi to fetch exact data for a UI component with just one API request.
MyMahi built their GraphQL API server using both graphql and graphql-tools libraries. Within a Lambda function, the GraphQL API server handles queries and mutations (i.e writes) to CockroachDB. Because CockroachDB is an ACID database, the GraphQL resolvers can use transactions that run concurrently as if they were isolated from each other. For example, one transaction cannot see the operations of another concurrent transaction. CockroachDB runs at SERIALIZABLE isolation, which is the safest level of isolation and protects against data anomalies that can happen at lower isolation levels (like, write skew).
Retriable Queries
Both retriable queries and auto-retry code has saved MyMahi time. Most of their code is written as is, and they don’t have to apply retry logic.
Rolling Upgrades
Being able to upgrade without taking the server offline has been a novel pleasure. In Postgres, downtime is required for upgrades which requires time to plan and execute appropriately.
JSONB Support
“This allows us to be a little lazy when we need to be,” says Stefan. The flexibility to temporarily store unstructured data in a relational database comes in handy sometimes.
The single biggest challenge for MyMahi is to give teachers and students a platform that is so easy to use, that even the most stubborn teachers will be willing to adopt MyMahi and include it in their work. Building with CockroachDB has simplified operations, and unlocked technical opportunities that have allowed developers to focus on making the application easy to use. MyMahi would not be where it is today if the developers had to maintain and scale Postgres.
Stay-at-home orders have increased the popularity of MyMahi and other digital platforms. If the increase in demand created bad user experiences it would have been hard for a startup to recover. But because of their forward thinking architecture and innovative philosophy MyMahi is prepared to meet increased demand and to scale across the ocean to Australia in 2021.
